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ABSTRACT 

A  description  is  given  of  a  program  designed  to  improve 
technology  transfer  and  utilization  in  a  classical  organization 
consisting  of  a  research  and  development  activity  and  several 
engineering  oriented  user  activities.   The  effectiveness  of  the 
technology  transfer  and  utilization  program  is  longitudinally 
studied.   Three  year  comparisons  of  several  parameters  are 
reported.   A  benefit  evaluation  decision  model  is  introduced  in 
order  to  improve  the  accuracy  of  the  cost/benefit  evaluation  of 
the  technology  transfer  and  utilization  program.   Using  this  model 
further  longitudinal  comparisons  are  made.   Finally  the  general 
usefulness  of  the  benefit  evaluation  decision  model  is  argued  by 
showing  that  for  each  dollar  of  investment  this  particular  program 
gave  a  present  value  return  of  $2.72  in  benefit.   Several  addi- 
tional comparisons  that  could  be  made  are  suggested. 


Technology  Transfer/  Introduction 

The  concept  of  "technology  transfer"  is  difficult  to  define 
since  its  meaning  seems  to  vary  with  the  audience  discussing  it. 
In  general,  however,  the  transfer  of  technology  differs  from  the 
usual  dissemination  of  scientific  knowledge  in  that  it  is  more 
concerned  with  the  usage  of  technological  information  obtained 
through  research/development  effort.   "Any  mechanism  developed 
for  transferring  technology  from  its  origin  to  its  usage  should 
be  directed  more  toward  suggesting  methods  and  areas  of  applica- 
tion than  toward  merely  publishing  scientific  results  in  technical 
documents  to  be  filed  in  technical  sections  of  depositories" 
(Doctors,  1969,  p.  56) . 

In  the  United  States  the  federal  laboratory  system  repre- 
sents a  vast  resource  of  science  and  technology,  with  over  469 
major  research  and  development  (R&D)  installations  (RCSG,  1973, 
p.  iii) .   These  agencies  have  been  producing  technological  reports 
at  about  50,000  per  year  (Olken,  1972,  p.  9).   Over  the  past 
decade  the  executive  branch  of  the  government  has  emphasized  the 
need  for  the  federal  government  to  actively  disseminate  this 
technology  to  the  public  and  private  sectors  in  an  effort  to 
increase  the  economic  benefit  of  the  information. 

At  present  the  federal  government  is  supporting  a  number 
of  technology  transfer  programs  in  several  agencies  and  depart- 
ments; however,  it  is  very  difficult  to  measure  their  results. 
"...  there  is  a  need  for  comprehensive  experiments  ...  which  would 
investigate  the  acquisition,  evaluation,  and  dissemination  of 


technical  information  and  the  measurement  of  its  use  after 
dissemination"  (Doctors,  1969,  pp.  7-8). 

Objective  of  the  Study 

The  objective  of  this  study  was  to  use  longitudinal  data 
to  evaluate  the  effectiveness  of  a  technology  transfer  and  utili- 
zation program  that  had  been  funded  in  1971  by  the  Naval 

1       .  . 
Facilities  Engineering  Command.    Specifically  the  program  under 

study  was  the  Facilities  Engineering  Support  Office  (FESO)  of  the 

2 

Navy's  Civil  Engineering  Laboratory  (CEL) . 

In  the  process  of  upgrading  the  evaluation  techniques  a 
benefit  evaluation  decision  model  was  introduced. 

A  second  objective  of  the  study  was  to  show  that  the  benefit 
evaluation  decision  model  is  a  useful  tool  in  terms  of  providing 
a  meaningful  method  of  quantifying  the  benefits  of  a  technology 
transfer  and  utilization  program. 

General  Background 

The  Naval  Facilities  Engineering  Command  (NAVFAC)  executes 
a  program  of  research,  development,  test  and  evaluation  (RDT&E) 
for  shore  facilities,  advance  base  and  amphibious  operations,  sea 
floor  structures,  environmental  control  and  those  aspects  of 
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weapon  systems  related  to  its  mission.   A  significant  portion  of 
the  emphasis  of  NAVFAC ' s  program  is  to  provide  RDT&E  which  will 
benefit  the  Navy's  shore  facilities  in  efficiently  and  effectively 
meeting  their  independent  missions.   NAVFAC ' s  link  to  the  shore 
facilities  is  primarily  through  the  Engineering  Field  Divisions 
(EFD's),  Public  Works  Centers  (PWC's),  Public  Works  Departments 
(PWD's),  and  its  construction  program  with  Officers  in  Charge  of 
Construction  (OICC's)  and  Resident  Officers  in  Charge  of  Construc- 
tion (ROICC's).   Figure  1  shows  these  relationships.   A  major  por- 
tion of  NAVFAC s  RDT&E  effort  is  assigned  to  CEL  in  the  form  of 
specific  research  projects. 

The  Civil  Engineering  Laboratory  (CEL)  is  the  principle 
research,  development,  test  and  evaluation  center  for  shore  and 
sea  floor  facilities  and  for  support  of  Navy  and  Marine  Corps 
construction  forces  (NCEL  Inst.  1970,  1972).   The  staff  of  CEL 
consists  of  approximately  320  personnel,  150  of  whom  are  profes- 
sional engineers. 

Funding  for  CEL's  FY' 74  program  exceeded  $13  million.   For 
FY '74  the  bulk  of  CEL's  efforts,  about  76%,  were  in  exploratory 
development  (applied  research) .   Roughly  1%  of  the  effort  was  in 
research,  and  the  balance,  about  23%,  was  in  advanced  engineering 
and  operational  systems  development  (RAP  BRIEF  1974,  p.  ii). 

NAVFAC ' s  Technology  Transfer  Program 

The  Navy's  RDT&E  funds  are  administered  by  a  division  of 
NAVFAC  which  has  the  responsibility  of  insuring  that  the  input  and 
output  of  R&D  information  is  transferred  between  all  levels  of  the 
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Figure  1.   Navy's  Civil  Engineering  Activity 


An  organizational  diagram  showing  the  flow  of 
technical  information  between  the  Naval  Facilities 
Command  Headquarters,  the  Civil  Engineering  Labora- 
tory and  the  various  engineering  field  activities. 


Navy  to  insure  that  the  maximum  benefit  from  R&D  expenditures  is 
obtained.   Two  specific  organizational  innovations  are  funded  by 
NAVFAC  in  an  effort  to  better  coordinate  the  flow  of  R&D  informa- 
tion from  and  to  the  field  operating  units  (see  Figure  2) : 

1.  The  Facilities  Engineering  Support  Office  (FESO) . 

2.  The  Engineering  Field  Division  Liaison  Billets. 

The  Facilities  Engineering  Support  Office 

The  FESO  organization  was  established  by  CEL  and  funded  by 
NAVFAC  to  perform  the  function  of  coordinating  services  and  com- 
munications related  to  RDT&E  assistance  to  Naval  shore  activities. 
The  specific  objective  of  this  program  was  to  provide  RDT&E  assist- 
ance to  Naval  shore  activities  by  having  CEL  perform  short-term 
services  to  determine  the  relative  value  and  suitability  of  new 
materials,  equipments,  processes  and  construction  or  maintenance 
procedures  (NCEL  Inst.  1971) . 

Conceptually,  the  FESO  is  in  a  liaison  position  and  as  such 
consists  at  the  present  time  of  one  civilian.   He  administers  the 
functions  of  the  office  by  coordinating  and  recording  the  flow 
of  information  between  the  field  units  and  the  specific  laboratory 
individuals  having  expertise  in  the  area  of  the  inquiry.   In  addi- 
tion, this  office  is  tasked  with  assuring  that  field  units  are 
knowledgeable  of  the  lab's  current  programs  and  the  availability 
of  the  FESO  service.   This  advertisement  function  is  pursued 
through  various  media  including  leaflets,  bulletins  and  site  visits 
A  24  hour  phone  service  is  also  maintained  by  FESO  to  handle  and 
record  incoming  calls  world-wide.   In  general,  CEL's  internal 
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Figure  2.   NAVFAC  Technology  Transfer  Program 


The  organizational  action  taken  by    CEL 
was  to  establish  a  Facilities  Engineering  Support 
Office  at  the  Civil  Engineering  Laboratory.   NAVIAC 
established  a  Liaison  Representative  at  each  of  the 
Engineering  Field  Divisions. 


guidelines  for  expenditures  of  the  funds  allotted  to  the  FESO 
provides  that  assistance  requests  involving  twenty  man-hours  or 
less  are  approved  at  Division/Department  level.   Requests  for  ser- 
vices involving  more  than  twenty  man-hours  (or  $500)  or  frequently 
recurring  requests  of  a  similar  nature  are  categorized  as  job 
order  requests.   These  requests  must  be  approved  by  the  technical 
director. 

The  Engineering  Field  Division  (EFD)  Liaison  Billets 

The  RDT&E  liaison  billet  was  intended  to  facilitate  communi- 
cation between  field  activities  and  NAVFAC  in  the  area  of  RDT&E 
and  to  serve  to  improve  and  expedite  the  evaluation  of  new  ideas, 
concepts,  procedures,  new  materials,  equipment,  and  to  insure  feed- 
back to  NAVFAC  for  further  dissemination  Navy -wide.   They  study 
technical  problems  unique  to  the  field  forces  and  transmit  to  NAVFAC 
proposals  for  research  effort  to  initiate  corrective  measures  (NAVFAC 
3900.6  of  13  July  1966).   They  also  serve  as  the  CEL  FESO  primary 
point  of  contact  at  each  of  the  EFD's. 

Evaluation  of  FESO  Benefits 
1.   Quantity  of  Requests 

The  number  of  requests  recorded  by  the  FESO  office  were  as 
follows:   FY1972  =  281,   FY1973  =  349   and   FY1974  =  396.   Under 
the  assumption  that  an  increasing  number  of  requests  indicated 
increasing  benefit  to  the  field  activities,  the  program  has  been 
expanding  well. 


Survey  techniques  and  details  of  each  year's  evaluation  may  be 
found  in  the  technical  reports  listed  in  the  reference  [Jolly  and 
Creighton,  1973,  Jolly  et  al.  Sept.  1974,  Jolley  et  al .  Dec.  1974] 


2.  Categories  of  Project  Requests 

Identifying  specific  categories  of  requests  by  project 
type  is  beneficial  for  determining  major  problem  areas  encoun- 
tered by  the  EFDs.    This  information  could  be  utilized  in  empha- 
sizing future  R&D  efforts  as  well  as  affecting  manpower  require- 
ments within  the  laboratory.   By  far  the  most  common  request  in 
each  year  studied  was  in  the  area  of  paint  and  coatings.   These 
accounted  for  between  18%  and  25%  of  the  total.   Other  problem 
areas  achieving  some  significance,  and  ranging  from  3%  to  12%, 
included  water  pollution,  classified  disposal,  structural,  corro- 
sion, concrete,  electronic,  pavement,  mechanical,  electrical,  and 
miscellaneous  pollution. 

3.  Means  of  Communication 

The  communication  link  to  and  from  CEL  was  considered  of 
vital  importance  to  the  program.   The  communication  system  must 
be  accessible  to  those  seeking  information  and  also  have  the 
ability  to  transmit  comprehensive  and  timely  data.   Survey  data 
showed  that  the  telephone  was  the  predominant  means  of  communicat- 
ing requests  from  the  Field  to  CEL.   The  use  of  the  telephone 
increased  from  60.2%  of  the  total  request  in  FY' 72  to  75.5%  in 
FY '74.   Contributing  factors  to  this  sizable  increase  were  the 
establishment  of  a  24  hour  answering  service  in  the  FESO  office 
and  emphasis  in  CEL  advertisement  of  their  accessibility  by  tele- 
phone.  The  survey  information  also  indicated  that  the  telephone 
was  a  major  means  of  communicating  responses  from  CEL  to  the  field, 
41.5%  in  FY' 72  and  54.8%  in  FY' 74.   It  should  be  noted,  however, 


that  the  data  used  to  calculate  the  above  percentages  was  based 
on  initial  request  and  initial  response.   In  many  cases,  letter 
and  message  follow  up  documentation  of  both  request  and  responses 
were  used. 

4.  Response  Time 

One  of  the  objectives  of  the  program  was  to  provide  a  rapid 
response  to  field  requests.   Results  of  measuring  this  parameter 
are  shown  as  Figure  3.   It  is  interesting  to  note  that  approxi- 
mately one-half  of  all  requests  were  answered  within  seven  days 
of  the  initial  request. 

5.  Utilization  of  Assistance 

An  important  measurement  of  the  program  effectiveness  is 
the  determination  of  the  degree  to  which  the  information  obtained 
from  CEL  was  used  by  the  requesting  activities.   Figure  4  is  a 
comparison  of  the  productive  versus  the  unproductive  contacts. 
The  results  are  expressed  in  percentage.   It  is  difficult  to 
attach  any  statistical  significance  to  the  relative  small  change 
that  was  observed  over  the  three  year  span. 

6.  Dollar  Benefit  of  Assistance 

The  measurement  of  the  dollar  benefit  of  the  NAVFAC  tech- 
nology transfer  program  involving  the  CEL  and  the  EFD's  was  con- 
sidered desirable.   The  measures  discussed  up  to  this  point  in 
this  paper  tend  to  indicate  that  the  program  was  working  well, 
but  they  are  not  expressed  in  dollar  benefits.   In  fact,  each 
years  evaluation  questionnaire  attempted  to  determine  dollar 
benefits  by  asking  the  requestor  to  quantify  within  specific 


Time 

Period  1  day  2  days  7  days  14  days  30  days 

FY'72  26  34  45  57  76 

FY'73  31  35  52  65  76 

FY' 74  27  37  57  63  75 


Figure  3.   Percent  of  Total  Requests 
Answered  Within  Time  Period 


Shown  is  the  cumulative  percent  of  requests  that  were 

answered  within  a  specific  time  period.   As  can  be  seen 

approximately  one-half  of  all  requests  were  answered 
within  seven  days. 


FY1972     FY1973     FY1974 
Productive  Output        72.8      82.8       79.0 
Unproductive  Output       27.2       17.2       21.0 


Figure  4.   Percentage  Utilization  of  the  Output 
Resulting  from  Requests 


It  is  quite  apparent  that  the  year  to  year  change  in 
productive  versus  unproductive  output  from  the  requests 
is  such  that  it  is  difficult  to  attach  any  statistical 
significance  to  the  small  change.   It  would  appear  that 
a  70  to  80  percent  utilization  over  time  can  be  expected 
The  results  shown  are  from  survey  data. 


ranges  the  estimated  five  year  operating  cost  of  individual 
projects,  with  and  without  the  assistance  from  CEL.   The  differ- 
ence between  the  with  and  without  assistance  cost  was  used  as  the 
benefit.   Using  this  method  the  results  reported  are  given  in 
Figure  5 . 

In  compiling  the  data  reported  in  Figure  5  it  was  observed 
that  only  a  small  fraction  of  the  requestors  that  completed  the 
questionnaire  were  willing  to  assign  a  dollar  value  to  their  use 
of  the  CEL  information.   For  example  in  FY1972  of  the  sample  of 
83/  only  12  or  14.5%  gave  dollar  values  such  that  dollar  benefit 
could  be  calculated.   It  was  somewhat  better  in  FY197  3,  the  sample 
was  93  and  28  or  30%  gave  dollar  values  such  that  dollar  benefit 
could  be  calculated.   Even  so,  it  was  felt  that  the  true  value  of 
the  benefit  was  not  being  evaluated  using  this  system.   The  next 
few  paragraphs  will  describe  in  detail  an  alternate  method  of 
obtaining  a  dollar  value  for  the  benefits  derived  from  the  trans- 
fer of  technology  by  the  CEL  to  the  EFD's. 

Benefit  Evaluation,  The  Problem 

Objectively  quantifying  the  benefits  of  the  CEL  technology 
transfer  effort,  as  indicated  in  the  previous  paragraphs,  was  at 
best  highly  subjective  and  lead  to  varying  results.   There  existed 
a  considerable  range  of  latitude  when  an  attempt  was  made  to 
quantify  the  dollar  value  of  the  benefit  derived  from  a  piece  of 
information. 

A  specific  recommendation  to  solve  a  particular  problem 
may  easily  be  quantified  if  it  will  reduce  out  of  pocket  expenditures 
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to  achieve  identical  results.   Factors  such  as  quality  of  output 
could,  however,  tend  to  cloud  even  this  type  of  calculation. 
Quantifying  benefits  derived  from  one  piece  of  information  which 
is  only  a  part  of  the  total  information  required  to  arrive  at  a 
decision  leaves  room  for  even  greater  subjectivity.   At  the  other 
end  of  the  scale,  quantifying  an  intangible  benefit  such  as  in- 
creased moral,  safety,  and  general  information  probably  is  the 
most  subjective  measurement  of  all  (Quinn,  1959,  p.  11). 

In  essence,  any  attempt  to  quantify  the  benefit  of  informa- 
tion is  necessarily  highly  subjective,  and  recognition  of  this 
fact  is  an  underlying  consideration  in  the  development  of  a  new 
approach  which  is  presented  here. 

Benefit  Evaluation,  The  Model 

It  was  determined  that  the  major  issue  in  evaluating  the 
benefit  from  a  technical  recommendation  was  the  categorization 
process.   A  system  was  needed  to  test  whether  or  not  a  benefit 
resulted  and  if  a  benefit  resulted  then  to  what  extent  the  recom- 
mendation was  responsible  for  the  final  benefit.   This  categoriza- 
tion may  be  accomplished  by  the  benefit  evaluation  decision  model 
shown  as  Figure  6  (Hendrickson  and  Fisher,  1974,  p.  33-47). 

Using  Figure  6  and  starting  with  an  answer  to  a  request  for 
technical  information  the  question  at  A   is,  "Did  the  answer  to 
the  request  for  technical  information  result  in  a  benefit  either 
tangible  or  intangible?"   If  the  answer  to  the  question  is  no, 
then  the  analysis  terminates  and  that  information  request  or  recom- 
mendation is  counted  as  having  zero  benefit.   If  the  answer  is  yes, 


Year 
FY' 7  2 
FY'  73 
FY '74 


Total 
Requests 
Number 

Sample 
Size 

Estimate 

from 

Sample 

Extrapolated 
to  Total 
Requests 

281 

29.5% 

$28,000 

$  94,915 

349 

26.6% 

$46,000 

$172,932 

396 

26.5% 

$77,000 

$290,566 

Figure  5.   Benefit  from  CEL  Assistance  Program 


Estimate  of  benefits  using  survey  questionnaire  data. 
This  does  not  include  extraordinary  benefits  reported, 
i.e.  FY'73  =  $15,000  and  FY  *  74  =  $187 , 000  . 
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Figure  6.  Benefit  Evaluation  Decision  Model 

Start  at  decision  A,  a  series  of  decisions  are  shown  that  make  it 
possible  to  evaluate  the  dollar  benefit  of  the  answer  to  a  technical 
question  supplied  to  an  engineering  organization  by  a  research 
laboratory. 


then  the  decision  next  in  order  on  the  decision  model  is   B,   "Is 
the  information  general  so  as  to  have  no  immediate  value  or  is 
the  information  specific  and  useful?"   If  the  information  is 
general  in  nature  it  may  be  filed  for  use  at  some  future  date 
and  the  present  value  of  the  dollar  benefit  is  assumed  to  be  the 
acquisition  cost.   In  contrast  if  the  information  is  specific, 
then  the  path  leads  to  decision  C.   At  decision   C   the  informa- 
tion must  fall  into  one  of  two  classifications,  either  partial 
information  for  a  decision  or  complete  information  for  a  decision 
When  the  information  is  partial,  that  is,  when  the  information  is 
only  part  of  the  total  information  used  to  arrive  at  a  decision, 
then  it  is  appropriate  and  necessary  to  determine  the  percent  of 
contribution  to  the  total  decision.   This  process  is  shown  as 
block   D   of  the  model.   When  the  information  is  substantially 
complete  and  self  contained  in  terms  of  influencing  the  decision 
then  one  hundred  percent  credit  for  the  benefit  is  allocated  to 
that  information  request.   Block   E   shows  this  assumption. 

Both  of  these  information  evaluation  channels  then  lead  to 
an  action  decision  block   F.   At  decision  point   F   there  are 
four  alternatives.   If  the  project  was  implemented  then  the  deci- 
sion route  is  to   G  where  it  is  necessary  to  determine  if  the 
results  were  estimable.   In  many  cases  a  specific  dollar  benefit 
can  be  assigned  at  this  point  of  the  analysis. 

A  project  may  be  delayed,  but  with  a  specific  plan  for 
future  implementation.   This  decision  is  represented  by   H.   Some 
future  action  benefits  can  be  estimated  in  terms  of  a  dollar 
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return,  others  are  more  intangible.   This  decision  is  made  at 
point   H. 

The  third  branch  of   F   is  for  those  projects  that  have  a 
delayed  plan  because  testing  is  required.   The  estimate  of  bene- 
fit is  based  on  the  assumption  that  if  the  tests  are  successful, 
then  the  project  will  be  implemented.   If  the  project  is  imple- 
mented there  should  be  a  resulting  benefit.   For  some  projects 
the  potential  benefit,  once  the  project  is  implemented  can  be 
estimated  (decision  point   J)   but  for  other  projects  the  benefit 
may  be  only  intangible  such  that  it  would  be  most  difficult  or 
perhaps  impossible  to  assign  a  specific  dollar  value  to  the  benefit, 

The  fourth  and  last  choice,  at  action  decision  block   F, 
is  delayed-plan-study-required.   This  is  similar  to  the  delayed- 
plan-test-required  except  that  a  feasibility  or  economic  study 
may  be  necessary  in  order  to  determine  the  worth  of  the  project. 
As  part  of  the  study  additional  tests  may  be  necessary  and/or 
desirable  in  order  to  reduce  the  risk  associated  with  implementing 
the  recommendation.   This  branch  has  the  lowest  probability  of 
resulting  in  a  dollar  benefit,  because  of  the  several  uncertain- 
ties that  must  be  considered.   If  it  should  be  implemented,  it, 
like  the  other  decisions  of  decision  block   F,   must  proceed  to 
the  next  level  of  decision  (decision  block   k)  in  order  to  deter- 
mine if  the  benefit  is  estimable  and  therefore  a  specific  dollar 
benefit  qt   if  the  benefit  is  intangible  and  therefore  difficult 
to  evaluate  the  benefit. 
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Perhaps  the  most  important  function  of  the  benefit 
evaluation  model,  is  to  provide  an  organized  system  of  classifi- 
cation or  categorization  of  the  benefits  to  answers  to  requests 
for  technical  information.   When  the  categorization  process  is 
completed,  then  it  is  possible  to  continue  toward  the  objective 
of  determing  the  benefit  derived  from  the  recommendation.   This 
requires  the  determination  of  the  percentage  of  contribution  of 
the  recommendation  to  the  final  product  decision,  the  assignment 
of  probabilities  to  the  likelihood  that  the  project  will  be  imple- 
mented and,  a  determination  of  the  dollar  base  for  each  benefit. 
These  issues  will  be  discussed  in  the  next  few  paragraphs. 

Determining  the  Coefficients  and  Dollar  Base 

To  use  the  benefit  evaluation  decision  model  it  is  necessary 
to  (1)  assign  values  to  the  factor,  information  contribution  per- 
centage (decision  D)  for  each  benefit  that  resulted  from  the  use 
of  technical  information  provided  by  CEL    to  (2)  assign  proba- 
bilities to  the  likelihood  of  implementation  of  expected  actions 
(decision  F) ,  and  to  (3)  determine  the  dollar  base  for  each  bene- 
fit that  resulted  from  the  use  of  technical  information  provided 
by  CEL.   A  discussion  of  each  follows: 
1.   Factor,  Information  Contribution  Percentage 

The  coefficient  for  the  factor,  information  contribution 
percentage,  was  assigned  to  each  case  on  an  individual  basis. 
The  total  information  available  in  order  to  make  a  decision  on 
implementing  a  project  was  reviewed  by  the  researchers.   The  fac- 
tor coefficient  was  selected  from  the  range  of  0.01  to  1.0  based 
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on  the  relative  effects  that  the  CEL  provided  information  had  on 
the  selection  of  the  most  beneficial  alternative  available.   Other 
considerations  in  the  selection  of  a  factor  coefficient  were  the 
availability  of  the  information  from  other  sources  and  the  rela- 
tive benefit  of  the  next  best  alternative  that  may  have  been 
selected  had  CEL  provided  information  not  been  available. 
2.   Implementation  Probabilities 

Responses  to  requests  for  technical  information  which  were 
considered  beneficial  (Decision  A)  were  classified  by  the  benefit 
decision  model  into  one  of  the  following  categories  which  are 
listed  in  descending  order  according  to  the  probability  of  imple- 


mentation of  the  information  provided: 
Final  Decision 


Information  has  been 
implemented 


Specific  plan  to 
implement 


Implement  if  tests  are 
successful 


Decision  Model 
Path 


A,  beneficial 

B,  specific  info. 

C,  evaluate 
E,  action 

G,  implement 


Probability  of 
Implementation 


value 


range 


1.0 


A, 
B, 
C, 
F, 
H,  planned  for  future    0.5 


A, 
B, 
C, 
F, 

J,  delay  plan  (test)     0.3 


0.4-0.6 


0.2-0.4 
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Implement  if  study  indicates 
benefits  are  probable 


General  information  not 
specific  to  a  project 


value  range 
A, 
B, 
C, 
F, 
K,  delay  plan  (study)    0.2      0.1-0.3 


A, 

B,  file  for  future  use 


The  probabilities  shown  were  based  on  the  experience  of  the 
researchers.   It  would  be  appropriate  to  adjust  these  probabilities 
depending  upon  the  history  of  the  organization  and/or  the  experi- 
ence of  the  researcher  using  the  benefit  evaluation  decision  model. 
3.   Dollar  Base  for  Model  Calculations 

Projects  with  dollar  savings  specifically  identified  on 
the  FY' 74  questionnaire  were  classified  as  estimable.   If  the 
identified  savings  were  of  the  one-time  type  (versus  recurring) , 
the  amount  so  identified  was  used  as  the  project  benefit  dollar 
base.   The  benefit  credited  to  CEL  assistance  in  such  a  case  was 
the  project  benefit  dollar  base  reduced  by  the  factor  for  informa- 
tion contribution  and  the  factor  for  implementation  probability. 
If  estimable  identified  savings  were  of  the  recurring  type,  the 
project  benefit  used  was  the  present  value  of  the  first  five  years 
of  savings.   A  present  value  factor  of  3.935  for  a  steady  cash  flow 
throughout  the  year  utilizing  a  10%  rate  of  return  was  used  as  the 
project  benefit  dollar  base.   Again,  the  benefit  credited  to  CEL 
assistance  was  the  project  benefit  dollar  base  reduced  by  the 
factor  for  information  contribution  and  the  factor  for  implementa- 
tion probability. 
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Beneficial  projects  resulting  from  CEL  supplied 
recommendations,  which  did  not  have  specifically  identified 
dollar  savings,  generally  fell  into  areas  where  the  benefit  was 
in  the  form  of  improved  operations,  better  morale,  increased 
safety,  improved  quality,  etc.   In  FY' 72  and  FY1 73  surveys, 
benefits  of  projects  of  this  type  were  left  unquantif ied.   With 
the  exception  of  responses  to  requests  which  fell  into  the  cate- 
gory, general  information,  each  response  to  a  request  should  have 
had  an  identifiable  benefit  even  though  it  was  not  readily 
quantifiable  in  terms  of  direct  dollar  savings.   Each  response  to 
a  request  for  technical  information  could  in  some  way  be  identi- 
fied with  an  implemented  or  proposed  project,  the  magnitude  of 
which  was  normally  relatively  easily  quantified. 

The  assumption  accepted  was  that  in  order  to  commit  funds 
to  a  project  a  decision  maker  must,  whether  he  realizes  it  or  not, 
expect  a  return  in  future  benefit  which  is  some  percentage  greater 
in  present  value  than  the  initial  outlay.   This  percentage  may 
vary  from  decision  maker  to  decision  maker  and  will  even  vary  with 
time  under  varying  circumstances  for  any  decision  maker.   Even 
though  the  investment  return  would  be  expected  to  vary,  it  is 
assumed  that  the  decision  maker  (in  any  organization) ,  who  decides 
on  implementation  of  a  project  based  on  supplied  technical  infor- 
mation, would  be  of  a  quality  such  that  the  results  of  their 
decision  over  the  long  run  and  on  the  average  would  yield  a  posi- 
tive benefit.   In  the  case  of  evaluating  the  output  of  CEL  assistance 
approximately  100  of  295  requests  for  information  in  the  FY' 74  sur- 
vey fell  into  this  unestimable  category.   The  benefit  then,  for  the 
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projects  in  this  category,  was  the  project  investment  reduced 
by  the  factor  for  information  contribution  percentage  and  the 
factor  for  implementation  probability. 

Application  of  the  Model  to  FY' 74  Data 

The  FY' 74  survey  included  295  requests.   FY1 74  question- 
naires were  completed  on  each  of  these  requests,  105  through  the 
six  EFD  RDT&E  representatives,  and  the  remainder  by  researchers 
at  the  Naval  Postgraduate  School,  Monterey,  California,  based 
on  telephone  interviews  with  the  original  requestor.   Of  these 
295  questionnaires,  233  indicated  that  the  requestor  considered 
that  he  had  received  beneficial  information.   The  remaining  62 
questionnaires  indicated  that  the  request  yielded  no  beneficial 
information.   However,  among  these  were  4  0  cases  which  indicated 
that  there  were  extraordinary  circumstances  indicating  that  these 
cases  should  not  be  included  as  zero  benefit  requests,  but  rather 
should  be  eliminated  from  the  sample  for  purposes  of  cost  benefit 
analysis  and  study.   Most  common  among  these  extraordinary  cases 
were: 

1.  The  request  was  merely  a  followup  on  a  previous 
request.  The  benefits  were  totally  included  on  the 
original  request  number  and  were  listed  on  this  one 
as  zero  to  avoid  double  counting. 

2.  The  lab  had  requested  additional  information  on 
the  problem  from  the  originator  but  the  information 
was  not  provided. 

Table  I  shows  the  tabulation  of  the  numbers  of  cases  and 
the  total  benefits  calculated  for  each  code  based  on  the  code 
numbers  assigned  by  Figure  7.   This  table  shows  three  figures 
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Benefit 
Code 


Probability  of 
Implementation 


Benefit  Estimate 


01  Information  has  been  implemented 

02  Same 

03  Specific  plan  to  implement 

04  Same 

05  Implement  if  tests  are  successful 

06  Same 

07  Implement  if  study  indicates  benefits   Estimable 

08  Same  Not  estimable 

09  General  information  not  specified  to   Acquisition  cost 

a  project 


Estimable 
Not  estimable 
Estimable 
Not  estimable 
Estimable 
Not  estimable 


Figure  7.   Benefit  Codes  According  to  Likelihood 
of  Deriving  Benefits 


Code  numbers  are  assigned  to  each  benefit  path 
of  the  model.   These  code  numbers  are  used  to 
identify  the  decision  combination  when  specific 
dollar  benefits  are  discussed. 


for  the  benefits  in  each  category.   The  three  figures  represent 
calculations  based  on  use  of  the  mean,  the  high,  and  the  low  values 
of  the  probability  of  implementation.   Figure  8  shows  the  plotted 
values  indicating  the  increasing  uncertainty  as  the  cumulative 
benefits  progress  to  include  more  subjective  estimates. 

A  Cumulative  Benefit  Curve 

The  curve,  Figure  8,  resulted  from  drawing  smooth  curves 
through  the  points  derived  from  Table  I.   They  represent  graphi- 
cally the  fact  that,  as  the  benefits  of  a  greater  percentage  of 
cases  in  the  sample  are  quantified  (a  greater  number  of  subjec- 
tive estimates  are  included) ,  the  total  estimate  of  cumulative 
benefits  becomes  more  subjective.   The  vertical  distance  between 
the  "high"  and  the  "low"  curves  at  any  point  on  the  horizontal 
scale  represents  the  range  within  which  the  estimate  could  reason- 
ably be  expected  to  vary  due  to  differing  personal  values  of 
estimators  or  decision  makers. 

Although  not  by  any  means  an  analytical  proof,  the  curve 
tends  to  intuitively  verify  the  applicability  of  the  model.   As 
intuitively  expected,  the  benefits  from  the  highly  intangible 
cases  are  less  than  those  from  the  more  tangible  ones,  as  indi- 
cated by  the  decreasing  slope  of  the  curves.   This  intuitive 
approach  is  further  strengthened  by  the  observation  that  a  deci- 
sion maker  will  generally  give  less  weight  to  the  more  intangible 
benefits  when  confronted  with  the  choice  of  whether  or  not  to 
spend  today's  very  tangible  dollars  for  future  benefits. 
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Figure  8.  CEL  Assistance  Dollar  Benefits  vs  Number  of 
Requests  with  Requests  Ordered  According  to 
Probability  of  Estimate 


This  figure  shows  the  curves  of  quantified  benefits 
utilizing  high,  mean  and  low  probability  of  imple- 
mentation.  The  slope  decreases  and  the  range  of 
accuracy  widens  as  greater  numbers  of  requests  in  the 
sample  are  quantified. 


Benefit 
Code 

Number  of 
Requests 

Value 

Low 

Mean 

High 

Not 

:  counted 

40 

-- 

— 

— 

Zero  benefit 

22 

0 

0 

0 

01 

27 

$254,361 

$254,361 

$254,361 

02 

51 

9,260 

18,525 

37,051 

03 

6 

15,399 

19,250 

23,099 

04 

38 

11,816 

29,540 

70,896 

05 

8 

41,690 

62,534 

83,379 

06 

8 

3,485 

10,455 

27,880 

07 

1 

236 

472 

708 

08 

12 

532 

2,130 

5,890 

09 

82 

unquantifie 

d 

295         $336,779   $397,267   $503,264 


Table  I.   Quantified  Benefits  for  FY '74  FESO  Operation 


Dollar  benefits  are  shown  according  to  benefit  code 
(see  Figure  7) .  The  dollar  benefits  are  calculated 
using  the  Benefit  Evaluation  Model. 


For  the  purposes  of  the  analysis  made  throughout  the 
remainder  of  this  study,  the  benefit  value  utilized  will  include 
Benefit  Codes  01  through  08  only  and  will  be  based  on  mean  factor 
values . 

A  Comparison  with  Past  Years '  Surveys 

The  FY' 74  survey  methods,  in  large  part,  have  resulted 
from  experience  gained  by  FY' 72  and  FY' 73  surveys.   Many  questions 
have  been  changed  to  some  degree  each  year  in  an  effort  to  obtain 
better  data.   One  disadvantage  of  the  continuous  change  is  the 
fact  that  the  figures  are  not  strictly  comparable  from  year  to 
year.   This  comparison  is  necessary,  however,  in  order  to  show 
trends  over  time. 

Data  from  the  FY' 74  survey  is  similar  to  that  of  the  earlier 
surveys  in  all  areas  except  the  dollar  value  estimates  of  benefits. 
The  major  differences  between  the  FY' 74  and  earlier  data  in  the 
area  of  quantifiable  benefits  is  shown  below: 
FY' 72  and  FY' 73  Surveys 

1.  Only  readily  estimable  cases  which  had  been  implemented 
were  included. 

2.  Of  the  total  Shore  Facilities  requests  received  4  0%  were 
surveyed. 

3.  Where  annual  savings  were  identified,  the  project  benefit 
was  calculated  as  five  times  the  annual  savings. 

4.  The  total  project  benefit  was  used  as  the  benefit  of  CEL 
assistance. 

FY' 74  Survey 

1.   Any  benefit  which  could  be  reasonably  quantified  was  used. 
Projects  with  future  implementation  potential  were  included 
and  the  benefits  thereof  were  reduced  by  applying  a  proba- 
bility factor. 
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2.  All  Shore  Facilities  requests  were  surveyed. 

3.  Where  annual  savings  were  identified,  the  present  value 
of  five-year  savings  was  used. 

4.  An  information-%  -  factor  was  applied  to  the  project 
benefit  to  arrive  at  the  benefit  contribution. 

The  dollar  benefit  figures  computed  by  FY1 74  survey  methods  must 
be  adjusted  to  be  comparable  to  the  earlier  survey  figures  for 
purposes  of  examining  trends. 

The  cases  in  the  FY '74  survey  which  were  classified  into 
Benefit  Code  01  are  only  partly  equivalent  to  the  cases  quantified 
in  the  FY'72/FY'73  surveys.   The  quantified  benefits  applicable 
from  this  code  totaled  $66,662.    Adjusting  this  figure  downward 
by  a  factor  of  0.4  to  $26,665  compensates  for  the  40%  sample  of 
the  FY'72/FY'73  surveys.   Further  adjusting  upward  by  dividing  by 
.37  to  $72,068  compensates  for  use  of  the  information-%  factor  in 
FY' 74.   Approximately  83%  of  the  readily  estimable  savings  in  the 
FY* 74  survey  were  of  the  annual  recurring  type.   The  resulting 
adjustment  for  use  of  the  present  value  factor  (472,068  x  3.9  x 
5  x  0.83)  yields  $76,688. 

By  a  similar  method,  the  FY' 72  and  FY1 73  survey  data  can 
be  adjusted  to  reflect  results  as  if  the  FY '74  method  was  applied 
to  all  three  years.   Figure  9  shows  a  graphical  comparison  of  all 
three  years  utilizing  both  the  FY'72/FY'73  and  the  FY '74  survey 
methods.   The  extraordinary  cases  have  been  eliminated  since  it 
is  inappropriate  to  include  them  in  projections  either  backward 


This  figure  excludes  one  project,  quantified  at  $187,000.   This 
project  is  considered  extraordinary  and  not  of  a  recurring  type 
and  should,  therefore,  not  be  included  at  this  point. 
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Figure  9.   CEL  Assistance  Benefit  Trends  (Thousands 
of  Dollars) 


The  bar  graph  shows  the  CEL  assistance  benefit  using 
(1)  estimates  of  the  requestors  and  (2)  using  esti- 
mates from  the  Benefit  Evaluation  Decision  Model. 


or  forward.   It  should  be  noted,  however,  that  over  the  long  term 
extraordinary  cases  of  this  type  may  constitute  a  significant 
portion  of  relaized  benefits. 

Benefit  Analysis  Using  Selected  Topics 

This  study  evaluated  only  295  requests  of  the  total  396 
that  the  FESO  office  recorded  during  FY' 74.   With  respect  to  the 
total  requests  the  analysis  represented  75%  of  the  CEL  assistance 
effort.   The  value  of  the  benefits  attributable  to  these  295  re- 
quests, as  measured  by  the  model  amounted  to  $395,269  (see  Table  1). 
Of  the  total  cost  of  the  CEL  assistance  for  FY' 74  only  the  portion 
of  the  total  cost  applicable  to  the  295  requests  would  be  appropriate 
for  a  cost/benefit  analysis.   These  costs  are: 

273  short-term  requests  $48,841 

22  job  order  requests  $66,346 

FESO  administrative  expenses,  proportioned 
as  to  total  assistance  requests  processed 

295/396  x  41,494  =  30,911  $30,911 

$146,098 

Evaluation  of  the  CEL  assistance  program  within  the  frame- 
work developed  above  indicates  that  there  was  a  present  value  return 
of  $2.72  for  every  dollar  spent  ($397 , 267/$146 , 093) . 

The  primary  purpose  of  the  CEL  assistance,  as  discussed  pre- 
viously was  to  provide  a  rapid  response  to  requests  for  assistance 
from  field  operating  units.   The  rapid  response  requests  were  re- 
corded separately.   If  these  short  term  rapid  response  requests  only 
were  considered,  then  the  appropriate  costs  should  include  direct 
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expenditures  for  the  273  short-term  requests,  plus  an  appropriate 
proportion  of  the  administrative  costs.   Expenditures  for  these 
short-term  requests  totaled  $77,447  (i.e.  $48,841  =  [273/396  x 
$41,494]).   The  dollar  value  of  the  benefits  from  these  273  requests 
was  $184,444.   The  return  of  the  CEL  assistance  program  for  these 
short-term  requests  was  $2.38  for  each  dollar  spent. 

When  the  dollar  benefit  of  each  project  can  be  determined 
it  is  then  possible  to  construct  charts  for  analysis  purposes.   For 
the  CEL  assistance  these  data  were  used  to  construct  the  following: 

1.  Comparison  of  costs  and  benefits  with  respect  to  project 
type,  i.e.  paints,  pavement,  pollution  corrosion,  etc. 

2.  Comparison  of  cost  and  benefits  by  benefit  code,  i.e. 
implemented,  specific  plan  for  future  implementation,  etc. 

3.  Comparison  of  cost  and  benefit  by  user  group,  i.e.  EFD's, 
PWC's,  OICC's,  etc. 

4.  Comparison  of  cost  and  benefit  by  originator,  i.e.  civilian 
engineer,  military  engineer,  technician,  scientist,  etc. 

Other  comparisons  meaningful  to  a  specific  organization  or 
evaluation  problem  should  be  obvious  depending  on  the  needs  and/or 
objectives  of  the  study. 

Summary  and  Conclusions 

This  longitudinal  study  of  a  classical  organization  with  a 
specific  structure  to  promote  technology  transfer  and  utilization 
has  carried  the  cost/benefit  analysis  beyond  the  usual  comparison 
of  number  of  requests,  response  time,  and  estimate  of  tangible 
benefits.   A  benefit  evaluation  decision  model  was  introduced 
that  provided  a  means  for  categorizing  technical  information  and/or 
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technical  recommendations.   The  model  considers  both  tangible 
and  intangible  benefits.   After  the  technical  information  and/or 
technical  recommendation  is  categorized  and  a  dollar  benefit  base 
assigned,  the  dollar  benefit  is  adjusted  (1)  according  to  the  per- 
cent of  influence  it  had  upon  the  final  technical  project,  and  (2) 
according  to  the  likelihood  that  the  project  would  be  implemented. 
Dollar  values  derived  from  the  model  are  then  used  to  determine 
cost/benefit  by  several  comparisons.   For  example,  when  the  model 
was  used  to  evaluate  requests  for  assistance  from  CEL,  technology 
transfer  and  utilization  program  showed  a  return  of  $2.72  for  each 
dollar  spent. 

Other  comparisons,  meaningful  to  a  particular  organization 
are  suggested. 

It  is  believed  that  this  study  demonstrates  that  it  is 
possible  to  meaningfully  quantify  in  dollars  a  significant  portion 
of  the  benefits  of  technical  information  and/or  technical  recom- 
mendations that  are  often  identified  as  intangible. 

This  benefit  evaluation  decision  model  should  be  particu- 
larly useful  in  evaluating  the  benefits  of  technology  transfer 
and  utilization  programs  in  other  organizations. 
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